Metrics of Resident Achievement for Defining Program Aims.

INTRODUCTION: Resident achievement data is a powerful but underutilized means of program evaluation, allowing programs to empirically measure whether they are meeting their program aims, facilitate refinement of curricula and improve resident recruitment efforts. The goal was to provide an overview of available metrics of resident achievement and how these metrics can be used to inform program aims. METHODS: A literature search was performed using PubMed and Google Scholar between May and November of 2020. Publications were eligible for inclusion if they discussed or assessed "excellence" or "success" during residency training. A narrative review structure was chosen due to the intention to provide an examination of the literature on available resident achievement metrics. RESULTS: 57 publications met inclusion criteria and were included in the review. Metrics of excellence were grouped into larger categories, including success defined by program factors, academics, national competencies, employer factors, and possible new metrics. CONCLUSIONS: Programs can best evaluate whether they are meeting their program aims by creating a list of important resident-level metrics based on their stated goals and values using one or more of the published definitions as a foundation. Each program must define which metrics align best with their individual program aims and mission.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Effect of Resident Physicians in a Supervisory Role on Efficiency in the Emergency Department.

INTRODUCTION: While patient throughput and emergency department (ED) length of stay (LOS) are recognized as important metrics in the delivery of efficient care, they must be balanced with the educational mission of academic centers. Prior studies examining the impact of learners on throughput and LOS when staffing directly with attending physicians have yielded mixed results. Herein we sought to examine the impact of a staffing model involving a supervisory resident "pre-attending" (PAT) on ED throughput and LOS, as this model offers a valuable educational experience for residents, but may do so at the expense of operational efficiency. METHODS: We retrospectively analyzed 26,702 unique patient encounters at a university-affiliated community ED between July 1, 2017-January 1,2019. The experimental group was comprised of patients seen primarily by midlevel providers, who staffed with a PAT, who subsequently staffed with an attending physician. The control group was comprised of patients seen by midlevel providers and staffed directly with attendings without a PAT. We used a parametric hazard model to analyze the effect of the presence of a PAT on service time, controlling for potential confounders including timing of presentation and patient demographics. RESULTS: The presence of a PAT is associated with a statistically significant increase in service time of five minutes (p = 0.006). Holding other variables equal, predicted service time in the experimental group was 173 minutes (95% confidence interval (CI), 171-176), while that for controls was 168 minutes (95% CI, 165-171). CONCLUSION: The presence of a PAT is associated with a statistically significant increase in service time, but the magnitude (five minutes) is likely operationally insignificant. The negligible increase in service time is offset by the benefit to residents' training. The results of this study may be helpful for residency programs considering the addition of a PAT shift structure.

Entrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment.

OBJECTIVE: Effective self-assessment is a cornerstone of lifelong professional development; however, evidence suggests physicians have a limited ability to self-assess. Novel strategies to improve the accuracy of learner self-assessment are needed. Our institution's surgical entrustable professional activity (EPA) implementation strategy incorporates resident self-assessment to address this issue. This study evaluates the accuracy of resident self-assessment versus faculty assessment across 5 EPAs in general surgery. DESIGN, SETTING, PARTICIPANTS: Within a single academic general surgery residency program, assessment data for 5 surgery EPAs was prospectively collected using a mobile application. Matched assessments (resident and faculty assessments for the same clinical encounter) were identified and the remainder excluded. Assessment scores were compared using Welch's t test. Agreement was analyzed using Cohen's kappa with squared weights. RESULTS: One thousand eight hundred and fifty-seven EPA assessments were collected in 17 months following implementation. One thousand one hundred and fifty-five (62.2%) were matched pairs. Residents under-rated their own performance relative to faculty assessments (2.36 vs 2.65, p < 0.01). This pattern held true for all subsets except for Postgraduate Year (PGY)2 residents and Inguinal Hernia EPAs. There was at least moderate agreement between matched resident and faculty EPA assessment scores (κ = 0.57). This was consistent for all EPAs except Trauma evaluations, which were completed by faculty from 2 different departments. Surgery resident self-assessments more strongly agreed with Surgery faculty assessments than Emergency Medicine faculty assessments (κ = 0.58 vs 0.36). CONCLUSIONS: Resident EPA self-assessments are equivalent or slightly lower than faculty assessments across a wide breadth of clinical scenarios. Resident and faculty matched assessments demonstrate moderate agreement.

Implementation of Entrustable Professional Activities into a General Surgery Residency.

OBJECTIVE: Concerns over resident ability to practice effectively after graduation have led to the competency-based medical education movement. Entrustable professional activities (EPAs) may facilitate competency-based medical education in surgery, but implementation is challenging. This manuscript describes 1 strategy used to implement EPAs into an academic general surgery residency. DESIGN, SETTING, PARTICIPANTS: A mobile application was developed incorporating 5 EPAs developed by the American Board of Surgery; residents and faculty from the Departments of Surgery, Emergency Medicine, and Hospital Medicine at a single tertiary care center were trained in its use. Entrustment levels and free text feedback were collected. Self-assessment was paired with supervisor assessment, and faculty assessments were used to inform clinical competency committee entrustment decisions. Feedback was regularly solicited from app users and results distributed on a monthly basis. RESULTS: One thousand seven hundred and twenty microassessments were collected over the first 16 months of implementation; 898 (47.8%) were performed by faculty with 569 (66.0%) matched pairs. Engagement was skewed with small numbers of high performers in both resident and faculty groups. Continued development of resident and faculty was required to sustain engagement with the program. Nonsurgical specialties contributed significantly to resident assessments (496, 28.8%). CONCLUSIONS: EPAs are being successfully integrated into the assessment framework at our institution. EPA implementation in surgery residency is a long-term process that requires investment, but may address limitations in the current assessment framework.

Misunderstanding the Match: Do Students Create Rank Lists Based on True Preferences?

INTRODUCTION: The "stable marriage" algorithm underlying the National Residency Match Program (NRMP) has been shown to create optimal outcomes when students submit true preference lists. Previous research has shown students may allow external information to affect their rank lists. The objective of this study was to determine whether medical students consistently make rank lists that reflect their true preferences. METHODS: A voluntary online survey was sent to third-year students at a single midwestern medical school. Students were given hypothetical scenarios that either should or should not affect their true residency preferences and rated the importance of six factors to their final rank list. The survey was edited by a group of education scholars and revised based on feedback from a pilot with current postgraduate year 1 residents. RESULTS: Of 175 students surveyed, 140 (80%) responded; 63% (88/140) reported that their "perceived competitiveness" would influence their rank list at least a "moderate amount. Of 135 students, 31 (23%) moved a program lower on their list if they learned they were ranked "low" by that program, while 6% (8/135) of respondents moved a program higher if they learned they were ranked "at the top of the list." Participants responded similarly (κ = 0.71) when presented with scenarios asking what they would do vs what a classmate should do. CONCLUSION: Students' hypothetical rank lists did not consistently match their true residency preferences. These results may stem from a misunderstanding of the Match algorithm. Medical schools should consider augmenting explicit education related to the NRMP Match algorithm to ensure optimal outcomes for students.

Curated Collections for Clinician Educators: Five Key Papers on Graduated Responsibility in Residency Education.

Introduction The Accreditation Council for Graduate Medical Education calls graduated responsibility "one of the core tenets of American graduate medical education." However, there is no clear set of resources for programs to implement a system of progressively increasing responsibilities for trainees. This project aimed to identify a set of high-yield papers on graduated responsibility for junior faculty members. Methods A study group of Academic Life in Emergency Medicine Faculty Incubator participants identified relevant literature on graduated responsibility via a comprehensive literature search and a call to the online medical education community; 59 total papers were identified. The most relevant and applicable were selected by the study group via a three-round modified Delphi process. Results Five key articles for junior faculty interested in implementing more robust graduated responsibility at their residency training program were selected and described here. Summaries of key points, along with considerations for faculty developers and relevance to junior faculty, are presented for each article. Conclusions The articles presented here provide a solid theoretical and practical basis for junior faculty to explore graduated responsibility. The five articles presented here provide the junior faculty with a toolkit to examine and improve their systems for assigning responsibilities in a graded fashion at their own institutions.

The Flipped Classroom: A Critical Appraisal.

INTRODUCTION: The objective of this study was to review and critically appraise the medical education literature pertaining to a flipped-classroom (FC) education model, and to highlight influential papers that inform our current understanding of the role of the FC in medical education. METHODS: A search of the English-language literature querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 296 papers related to the FC using either quantitative, qualitative, or review methods. Two reviewers independently screened each category of publications using previously established exclusion criteria. Eight reviewers then independently scored the remaining 54 publications using either a qualitative, quantitative, or review-paper scoring system. Each scoring system consisted of nine criteria and used parallel metrics that have been previously used in critical appraisals of education research. RESULTS: A total of 54 papers (33 quantitative, four qualitative, and 17 review) on FC met a priori criteria for inclusion and were critically appraised and reviewed. The top 10 highest scoring articles (five quantitative studies, two qualitative studies, and three review papers) are summarized in this article. CONCLUSION: This installment of the Council of Emergency Medicine Residency Directors (CORD) Academy Critical Appraisal series highlights 10 papers that describe the current state of literature on the flipped classroom, including an analysis of the benefits and drawbacks of an FC approach, practical implications for emergency medicine educators, and next steps for future research.

Assessing Residency Applicants' Communication and Professionalism: Standardized Video Interview Scores Compared to Faculty Gestalt.

INTRODUCTION: The Association of American Medical Colleges has introduced the Standardized Video Interview (SVI) to assess the communication and professionalism skills of residency applicants to allow a more holistic view of applicants beyond academic performance. Initial data suggests scores are not correlated with academic performance and provide a new measure of applicant attributes. It is not currently known how the SVI compares to existing metrics for assessing communication and professionalism during the interview process. METHODS: Applicants to the University of Wisconsin Emergency Medicine Residency program were invited and interviewed without use of the SVI scores or videos. All faculty interviewers were blinded to applicants' SVI information and asked to rate each applicant on their communication and professionalism on a scale from 1-25 (faculty gestalt score), analogous to the 6-30 scoring used by the SVI. We transformed SVI scores to our 1-25 system (transformed SVI score) for ease of comparison and compared them to faculty gestalt scores as well as applicants' overall score for all components of their interview day (interview score). RESULTS: We collected data for 125 residency candidates. Each applicant received a faculty gestalt score from up to four faculty interviewers. There was no significant correlation of SVI scores with faculty gestalt scores (Spearman's rank correlation coefficient [rs] (123)=0.09, p=0.30) and no correlation with the overall interview score (rs(123)=0.01, p=0.93). Faculty gestalt scores were correlated positively with interview scores (rs(123)=0.65, p<0.01). CONCLUSION: SVI scores show no significant correlation with faculty gestalt scores of communication and professionalism. This could relate to bias introduced by knowledge of an applicant's academic performance, different types of questions being asked by faculty interviewers, or lack of uniform criteria by which faculty assess these competencies. Further research is needed to determine whether SVI scores or faculty gestalt correlate with performance during residency.

Maggots, Mucous and Monkey Meat: Does Disgust Sensitivity Affect Case Mix Seen During Residency?

INTRODUCTION: Emergency physicians encounter scenarios daily that many would consider "disgusting," including exposure to blood, pus, and stool. Physicians in procedural specialties such as surgery and emergency medicine (EM) have lower disgust sensitivity overall, but the role this plays in clinical practice is unclear. The objective of this study was to determine whether emergency physicians with higher disgust sensitivity see fewer "disgusting" cases during training. METHODS: All EM residents at a midsize urban EM program were eligible to complete the Disgust Scale Revised (DS-R). We preidentified cases as "disgust elicitors" based on diagnoses likely to induce disgust due to physician exposure to bodily fluids, anogenital anatomy, or gross deformity. The "disgust elicitor" case percent was determined by "disgust elicitor" cases seen as the primary resident divided by the number of cases seen thus far in residency. We calculated Pearson's r, t-tests and descriptive statistics on resident and population DS-R scores and "disgust elicitor" cases per month. RESULTS: Mean DS-R for EM residents (n = 40) was 1.20 (standard deviation [SD] 1.24), significantly less than the population mean of 1.67 (SD 0.61, p<0.05). There was no correlation (r = -0.04) between "disgust elicitor" case (n = 2191) percent and DS-R scores. There was no significant difference between DS-R scores for junior residents (31.1, 95% confidence interval [CI], 26.8-35.4) and for senior residents (29.0, 95%CI, 23.4-34.6). CONCLUSION: Higher disgust sensitivity does not appear to be correlated with a lower percentage of "disgust elicitor" cases seen during EM residency.